The Merits of Externally Invalid Survey Experiments

 

Gustavo Diaz
McMaster University
gustavodiaz.org

 

Slides: talks.gustavodiaz.org

Limitations

“Future research should confirm if our findings generalize…”

  • …with a representative sample
  • …in other countries
  • …beyond the survey setting
  • …when using behavioral outcomes

Usual workflow

  1. Consider resource/ethical/practical limitations

  2. Conduct survey experiment with convenience sample

  3. Wave hands about external validity

Goal

  • Move beyond limitations approach

  • Should we ever implement an externally invalid survey experiment on purpose?

  • Identify what makes external invalidity desirable

  • Challenge: Different kinds of external (in)validity

External validity concerns

Type Concern
Sample Does this apply to a different population?
Treatments Do they resemble real-world phenomena?
Outcomes Do they reflect actual behaviors?
Contexts Does this apply in a different setting?

External validity concerns

Type Concern
Sample Does this apply to a different population?
Treatments Do they resemble real-world phenomena?
Outcomes Do they reflect actual behaviors?
Contexts Does this apply in a different setting?

External validity concerns

Type Concern
Sample Does this apply to a different population?
Treatments Do they resemble real-world phenomena?
Outcomes Do they reflect actual behaviors?
Contexts Does this apply in a different setting?

External validity concerns

Type Concern
Sample Does this apply to a different population?
Treatments Do they resemble real-world phenomena?
Outcomes Do they reflect actual behaviors?
Contexts Does this apply in a different setting?

External validity concerns

Type Concern
Sample Does this apply to a different population?
Treatments Do they resemble real-world phenomena?
Outcomes Do they reflect actual behaviors?
Contexts Does this apply in a different setting?

Examples

Samples

Treatments

Outcomes

Contexts

Conclusion